Proceedings  of  the  2014  Winter  Simulation  Conference 

A.  Tolk,  S.  Y.  Diallo,  I.  O.  Ryzhov,  L.  Yilrnaz,  S.  Buckley,  and  J.  A.  Miller,  eds. 


USING  SIMULATION  TO  EXAMINE  LIVE-FIRE  TEST  CONFIGURATIONS 


Raymond  R.  Hill 
Darryl  Ahner 


AFIT/ENS 

2950  Hobson  Way,  Bid  641 
Wright-Patterson  AFB,  OH  45433,  USA 


Michael  J.  Garee 


AFOTEC  Det  1/DTS 
187  N.  Flightline  Rd„  Bid  1806 
Edwards  AFB,  CA  93524,  USA 


ABSTRACT 

Man-Portable  Air-Defense  System  (MANPADS)  missiles  are  threats  to  military  aircraft.  Analytical  models 
arc  used  to  help  design  military  aircraft  to  survive  a  variety  of  attacks,  including  those  from  Man-Portable 
Air-Defense  Systems.  These  models  need  accurate  fragment  capture  data  consisting  of  the  fragment  size 
and  velocity  resulting  from  weapon  detonation.  Accurate  data  require  accurate  testing  which  in  turn  requires 
effective  test  design  infrastructure.  We  model  this  test  infrastructure.  MANPADS  missiles  are  detonated 
within  test  arenas  that  have  make-screens  placed  on  the  arena  walls  to  capture  fragment  impact  data.  Our 
model  mimics  the  test  process  and  provides  a  quantitative  metric  with  which  to  examine  and  compare  test 
arena  configurations.  We  overview  our  model  and  quality  metric  and  offer  a  case  study  in  which  these  arc 
used  to  find  a  robust  arena  make-screen  configuration. 

1  INTRODUCTION 

Military  aircraft  systems  arc  designed  to  survive  being  attacked.  The  weapons  deployed  against  military 
aircraft  can  be  particularly  lethal.  Since  military  operations  often  require  military  aircraft  to  operate  in 
areas  of  high  threat,  these  systems  must  be  designed  to  avoid,  counter  and,  if  hit,  survive  in  these  high 
threat  areas.  Aircraft  survivability  is  the  general  label  applied  to  research,  development  and  design  efforts 
focused  on  ensuring  our  military  aircraft  systems  meet  these  threats. 

A  particularly  troublesome  threat  arc  Man-Portable  Air-Defense  System  (MANPADS)  missiles.  These 
missiles  arc  small,  portable  and  lethal,  shoulder  tired  and  best  suited  for  ground  targets  and  low-flying 
aircraft  (Bureau  of  Political-Military  Affairs  2011).  Their  size  makes  them  easy  to  transport  and  hide 
and  thus  they  are  quite  popular  in  the  more  unsavory  activities.  There  arc  an  estimated  500,000-750,000 
MANPADS  missiles  currently  stockpiled  (Schroeder  2007). 

MANPADS  analytical  models  require  good  data  on  fragment  dispersal  and  fragment  velocity  at  impact. 
Such  data  arc  obtained  via  live-fire  testing  which  involves  detonating  a  test  missile  in  a  specially  constructed 
test  arena.  Figure  1  depicts  one  such  arena  with  the  missile  statically  loaded.  The  walls  of  the  test  arena 
provide  a  means  to  record  fragment  impact  times  (for  fragment  velocity  estimates)  and  to  recover  the 
fragments  (weighed  to  calculate  residual  mass).  Fragment  impact  times  arc  recorded  when  the  fragments 
impact  specifically  designed  panels  mounted  on  the  arena  walls.  The  panels,  or  make-screens,  arc  of 
varied  sizes  and  arc  arranged  on  an  arena  wall  based  on  expert  opinion.  The  panel  arrangement  and  the 
effectiveness  of  that  arrangement  arc  the  focus  of  our  research. 

2  A  FRAGMENT  TEST  SIMULATION  MODEL 

Garee  et  al.  (2014)  describe  the  design  of  our  fragment  capture  simulation  for  MANPADS  testing.  We 
provide  a  general  overview  here.  The  model,  developed  in  MATLAB,  simulates  the  dispersal  of  missile 
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Figure  1:  A  MANPAD  missile  centered  in  a  fragment  capture  arena  (Czarnecki  et  al.  2011).  Rectangular 
patterns  on  the  walls  show  the  edges  of  the  make-screens  mounted  on  the  test  arena  walls. 

fragments.  This  dispersal  pattern  is  based  on  some  user-defined  distribution,  data  from  other  models,  or 
even  from  live-fire  test  events.  Fragment  trajectories  are  calculated  and  evaluated  for  impact  against  the 
arena  test  walls. 

In  actual  tests,  the  arena  walls  are  covered  with  make-screens.  These  make-screens  vary  in  size,  are 
tethered  to  data  channels,  and  are  used  to  collect  the  test  data  (see  Figure  1).  As  fragments  impact  the 
screens,  the  time  of  impact  is  recorded  over  the  data  channel.  This  impact  time  is  used  to  estimate  the 
fragment  velocity.  After  the  test  is  complete,  the  fragments  are  physically  removed  from  the  arena  walls, 
measured  and  counted  for  each  make-screen. 

The  recording  fidelity  of  each  make-screen  prohibits  assigning  impact  times  to  each  specific  fragment; 
only  the  time  of  an  impact  is  recorded.  Thus,  the  velocity  estimate  assigned  for  each  frament  is  the  average 
velocity  for  all  impacts  against  that  make-screen. 

The  simulation  mimics  this  test  process.  Data  input  defines  the  make-screen  configuration  implemented. 
Fragment  impacts  are  then  assigned  to  the  make-screen  impacted.  Each  fragment  impact  time  is  used  to 
calculate  the  average  velocity  for  all  impacting  fragments  on  that  make-screen.  The  difference  with  the 
simulation  is  that  the  actual  fragment  velocities  modeled  are  known  (assigned  at  fragment  creation)  which 
provides  a  means  to  develop  assessment  metrics  for  a  given  make-screen  configuration.  Such  a  metric 
can  be  used  to  design  more  effective  test  arenas  and  in-turn  provide  higher  quality  test  data.  Figure  2 
shows  a  simulation  result  with  fragments  scattered  against  the  arena  walls  having  a  simple  make-screen 
configuration  assigned  to  it. 

3  A  QUALITY  METRIC  FOR  ARENA  CONFIGURATIONS 

Garee  (2014)  defined  the  characteristics  needed  in  a  metric  for  arena  configuration  comparisons  and 
developed  such  a  metric.  This  metric  is  based  on  mean  absolute  velocity  error  (MAVE)  for  a  fragment, 
which  is  then  aggregated  into  a  measure  for  the  make-screen  and  finally  aggegated  into  a  measure  for  the 
arena  configuration.  For  ns  make-screens,  define  a  fragment  true  velocity  as  v/  and  its  estimated  velocity 
as  vs,  then  calculate  MAVE  for  screen  ,v  as 

1 

MAV Es  =  ,  £  |v/-v,|  s  =  l,...,ns.  (1) 

11  /=l 
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Make-screens  and  fragments  in  arena  coordinate  system 


Figure  2:  Sample  arena  with  fragments.  Twenty  fragments  are  displayed  on  each  wall. 
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This  value,  MAVES  is  normalized  to  Qs  =  1  —  MAV Es / MAV E*  where  MAVE*  is  the  worst-case  value. 
The  score  for  the  arena  configuration  is  then  defined  as 


Qa 


(2) 


where  Sactive  is  the  set  of  screens  receiving  a  fragment  impact  and  M  =  \Sactive\.  Larger  Q  scores  reflect 
more  effective  arena  configurations.  See  Garee  et  al.  (2014)  for  more  details  on  the  metric  development. 


4  FINDING  ROBUST  CONFIGURATIONS 

It  should  be  obvious  that  any  fragment  pattern  will  be  best  served  by  a  particular'  make-screen  arrangement 
tailored  to  that  fragment  pattern.  The  analytical  challenge  is  to  find  the  make-screen  arrangement  that  does 
the  best  job  against  any  of  the  many  potential  fragment  patterns  to  which  it  may  find  itself  subjected. 

The  current  simulation  model  is  quite  useful  for  assessing  make-screen  arena  configurations  against 
various  fragment  pattern  as  both  the  make-screen  configuration  and  fragment  pattern  are  user-defined.  The 
following  example  from  Garee  (2014)  demonstrates  how  to  find  robust  configurations. 

Hill  and  McIntyre  (2000)  worked  with  a  large-scale  linear  program  (LP)  and  defined  a  “robust  solution” 
as  the  particular-  military  force  structure  suggested  by  the  LP  as  providing  some  best  overall  solution  evaluated 
against  some  set  of  defense  planning  scenarios.  In  the  current  context,  we  seek  some  make-screen  arena 
configuration  that  provides  some  best  overall  performance  against  a  range  of  fragment  patterns.  The  current 
simulation  requires  the  make-screen  configuration  as  input  to  the  simulation  so  the  search  for  a  robust 
make-screen  configuration  requires  using  a  suite  of  configurations.  For  the  example  we  limit  the  arena  size 
to  a  single  wall  to  ease  the  presentation  and  do  not  limit  the  number  of  make-screens  that  can  be  placed 
on  that  wall. 

Figure  3  provides  the  six  varied  configurations  used  in  the  example.  These  configurations  were  manually 
generated  and  intended  to  provide  some  range  in  the  configuration  options.  These  six  configurations  also 
vary  the  complexity  of  the  configuration  realizing  that  more  make-screens  require  more  data  channels. 
Figure  4  shows  the  four  sample  fragment  distribution  patterns  used  for  the  robust  analysis.  These  patterns 
are  not  typical  of  live-fire  test  results  nor  do  they  come  from  existing  fragment  pattern  models.  The  patterns 
were  generated  to  provide  a  varied  sample  of  patterns.  No  random  fragment  patterns  are  used. 

Each  of  the  patterns  were  tested  against  each  of  the  make-screen  configurations  and  the  quality  scores 
compiled.  Those  results  are  provided  in  Table  1 .  We  note  that  configuration  D  does  the  poorest  across  the 
board.  This  configuration  uses  the  least  number  of  make-screens,  so  in  terms  of  resources  would  probably 
be  the  cheapest,  but  the  large  make-screen  size  means  more  fragments  impact  each  make-screen  thus 
causing  the  average  velocity  calculation  to  involve  more  fragments  and  be  applied  to  that  larger  number 
of  fragments.  The  best  performing  configurations,  in  terms  of  mean  quality  score,  are  configurations  A 
and  F.  However,  we  note  that  the  performance  of  configuration  A  has  more  variability  than  we  see  in 
the  performance  of  configuration  F.  Based  on  the  overall  consistent  performance  and  the  highest  overall 
mean  quality  score,  this  example  finds  configuration  F  to  be  the  most  robust  of  the  make-screen  arena 
configurations  considered. 


5  CONCLUSIONS  AND  FUTURE  RESEARCH  DIRECTIONS 

This  paper  summarizes  a  simulation  model  of  MANPADS  missile  detonation  and  presents  the  effectiveness 
measure  developed  to  assess  arena  make-screen  configuration  performance  versus  some  missile  fragment 
dispersion.  We  present  as  a  case  study  an  example  where  the  model  can  be  employed  to  find  a  robust 
make-screen  configuration. 

Our  work  is  still  preliminary  and  evolving.  The  current  research  represents  just  the  initial  effort. 
Subsequent  work  will  enhance  the  model  to  consider  full  arena  enclosures  and  will  consider  other  types 
of  make-screens. 
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Configuration  A  Configuration  B 


Configuration  C  Configuration  D 


Configuration  E  Configuration  F 
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Figure  3:  Six  make-screen  configurations  for  an  8  x  8  unit  wall.  Each  configuration  is  arbitrarily  created 
from  a  limited  pallet  of  make-screen  sizes.  The  smallest  make-screen  is  a  one-unit  square. 
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Distribution  1 


Distribution  2 


Distribution  3 


Distribution  4 


Figure  4:  Fragment  distribution  patterns  used  for  robust  make-screen  configuration  analysis. 
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Table  1:  Arena  data  quality  scores  for  various  pairings  of  arena  configurations  and  fragment  distributions. 
The  highest  score  for  each  distribution  is  in  boldface. 
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Fragment  Distribution 

Qa 

1 

2 

3 

4 

Mean 

A 

0.7511 

0.6788 

0.6979 

0.7652 

0.7232 

B 

0.7628 

0.6745 

0.6946 

0.6879 

0.7050 

C 

0.7608 

0.6923 

0.7099 

0.7110 

0.7185 

D 

0.5828 

0.6006 

0.4798 

0.5511 

0.5536 

E 

0.7095 

0.6757 

0.6833 

0.6930 

0.6904 

F 

0.7825 

0.7101 

0.7249 

0.7292 

0.7367 

Mean 

0.7249 

0.6720 

0.6650 

0.6896 

Our  research  challenge  is  how  to  generate  reasonable  make-screen  configurations.  The  configurations 
used  in  our  example  are  manually  developed.  Computer  generation  requires  considering  make-screen  sizes, 
shapes,  placement  on  the  wall  and  the  number  of  each  placed  on  the  arena  walls.  This  is  not  an  easy 
problem.  However,  such  a  generation  approach  can  be  exploited  in  a  simulation-based  optimization  search 
process  to  allow  the  model  to  find  robust  make-screen  arena  configurations. 
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